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I. ABOUT THIS GUIDEBOOK 



This Guidebook is designed to help state educational agencies (SEAs) create an effective system to 
evaluate state- approved supplemental educational service (SES) providers. This Guidebook will help 
readers to determine evaluation measures, identify possible evaluation methodologies, and address 
the technical and practical considerations associated with an evaluation. Although this Guidebook is of 
primary interest to states, it can also help school districts and SES providers understand their roles in 
the evaluation process. 

The first edition of this Guidebook was published in spring 2005 by the Supplemental Educational Ser- 
vices Quality Center, which was operated by the American Institutes for Research® (AIR®) from 
2003- 2005. Followingthe publication of the first edition of the Guidebook, much experience was 
gained and lessons were learned through the initial evaluation studies that the present authors and 
other groups conducted. At the same time, stakeholders in the evaluation process (e.g., policy makers, 
regulators, educators, providers, and evaluators) have gained an increased understanding of expec- 
tations, challenges, and potentially useful evaluation models. This revised edition of the Guidebook 
reflects these advances in the field. For example, we now propose, even more strongly than we did 
previously, that evaluation designs should supplement the mandatory examination of student achieve- 
ment outcomes with performance dimensions addressing customer satisfaction and service delivery, 
including provider efforts to promote regular student attendance. Like the first edition, this Guidebook 
explains these performance domains and describes possible evaluation measures and designs that 
states can use to assess each domain. This version goes one step further by proposing strategies for (a) 
"front-end" communications with providers about evaluation reguirements and expectations, and (b) 
end-of-year decisions about provider status (continuation or removal) based on the evaluation results. 

Overview of Supplemental Educational Services 

SES is a provision of Title I of the Elementary and Secondary Education Act (ESEA), as reauthorized by 
the No Child Left Behind (NCLB) Act. Through SES, students from low- income families are eligible to 
receive extra academic assistance if they attend a Title I school that has not made adequate yearly prog- 
ress (AYR) for at least 3 three years (i.e., a school that is at least in its 2nd year of "school improvement" 
status) . A variety of organizations are eligible to provide SES after receiving formal approval from the 
state, such as: 

# For-profit companies 

# Nonprofit groups 

# Local oornmimityprograrris 

# Colleges or universities 

# National organizations 

# Faith- based groups 

# Private and charter schools 

# Public schools and districts not identified as in need of improvement^ 

Roles of States and Districts 

NCLB and the U.S. Department of Education (ED) require states and districts to fulfill distinct roles in 
SES implementation. Each state is charged with identifying schools that do not make AYP and schools 
that have been identified as in need of improvement, corrective action, or restructuring. Each state de- 
velops its own SES provider application and criteria. 

1 

The U.S. Department of Education (ED) recently granted vaivers to a select number of districts identified for improvement 
to allow those districts to provide SES. For more information about this pilot, go to: http://www.ed.gov/news/pressreleases/ 
2006/ 07/ 0762006a.html. 



4 Evaluating SES Providers 



States must also monitor each approved provider. States are required to evaluate provider effectiveness 
after at least 2 years, but they may choose to monitor providers more frequently.^ The standards used 
to evaluate providers should be consistent for each provider and should be aligned with the criteria set 
forth in the state's SES provider application. According to NCLB, at a minimum, states must remove 
providers from the approved list if they fail to: 

# Increase students' achievement for 2 consecutive years. 

# Provide services consistent with applicable federal, state, and local health, safety, and dvil 
rights requirements. 

States may collect and report additional information about providers, including parent and student 
satisfaction with services. The SEAs may also request that districts help monitor SES providers. 

Districts are charged with determining vdiich students in Title I schools identified as in need of im- 
provement are eligible for SES.^ SES is available to low- income students in those schools— generally, 
those students eligible for free or reduced- price lunch. Once the district determines which students are 
eligible, it notifies fa mili es at least once per year that their children qualify for SES. The district must 
also provide families with information about local SES providers to help them choose a provider. If 
fa mili es ask for assistance, the district must help them choose a SES provider. If fa mili es' demands for 
SES exceed the available funds, districts must then give priority to the lowest- achieving eligible stu- 
dents. Once fa mili es choose a provider, then the district contracts with the provider and pays for its 
services. When establishing contracts, districts and providers should work together to clearly indicate 
requirements and expectations of the SES evaluation, particularfy with regard to how "effectiveness in 
increasing student achievement" will be determined. 



Working Together: Connecting State and District Efforts to Improve Evaluation 

States and districts bear distinct and critical responsibilities for implementing SES. Experi- 
ence suggests that regular communication and collaboration can help states and districts 
carry out these responsibilities more effectively. States are responsible for evaluating SES 
providers. States may ask school districts for assistance, particularfy vdien data are needed 
regarding: (a) student participation in SES; (b) student attendance; and (c) experiences and 
satisfaction with provider services by district SES liaisons/ coordinators, parents, and teach- 
ers and principals from participating schools. In such instances, states should ensure that the 
district's role is appropriate and unbiased, because school districts may also be SES providers. 
States and districts can also collaborate to ensure that district contracts with providers estab- 
lish procedures that states need to gather data and evaluate providers. Additionally, many of 
the evaluation options discussed in this Guidebook require consistent, statewide data gather- 
ing and reporting mechanisms. To ensure consistency and availability of required data for 
monitoring and evaluating SES, states are stron^y encouraged to implement a statewide data 
collection and compilation system. (For more guidance on data collection options, consult the 
Technology and Database Considerations box on page 15.) 



o 

^States are enc»uraged to monitor providers at least annually. 

a Title I school does not make AYP for 2 consecutive years, it must offer aU students in that school the opportunity to 
transfer to another pubhc school that is not in need of improvement. Currently, ED is exploring the viabihty, through waivers, 
of offering states flexibility in choosing the order of implementing the transfer and SES options to gualifying students. Addi- 
tional information about NCLB's choice options is available from ED's Web site: http:/ / www.ed.gov/ parents/ schools/ choice/ 
definitions.html. 






Suggested Strategies for States 5 



II. GUIDING QUESTIONS FOR EVALUATING SUPPLEMENTAL EDUGATIONAL 
SERVIGE PROVIDERS 



To effectively monitor SES providers, states must develop a system that will offer valid and highly us- 
able data to evaluate the impact of each SES provider's services. States are encouraged to consider a 
provider's impact in three areas: 

L Effectiveness. Did the provider increase student achievement in reading/ language arts or math? 

2. Customer satisfaction. Are parents of students who received SES satisfied? 

3. Service delivery and compliance. Did the provider comply with applicable state and district 
laws and contractual procedures associated with the delivery of SES? 

It is important to remember that providers are expected to demonstrate effectiveness across all of the 
types of students served, including English language learners (ELLs) and students with disabilities and 
a variety of learning needs. However, as we will discuss in the following sections, a realistic constraint 
to evaluating such effectiveness meaningfully and validly is that individual providers and grade levels of 
interest may have only small sample sizes for test scores necessary for analysis. Aocordin^y, the orien- 
tation that we recommend throughout the Guidebook is a pragmatic one based on the following prem- 
ises: 



# Evaluation is useful for decision making only when based on valid data. 

# SES evaluations will be restricted in any given year by the number of participating students 
for which appropriate (valid) achievement data are available. 

# Each year that achievement data are collected, the reliability of information about providers 
overall and for different student subgroups increases. 

# Over time, the ability to make confident and scientifically valid decisions about provider ef- 
fectiveness will increase. 

States may also monitor SES providers' instructional programs to determine whether the services deliv- 
ered match those described in the providers' applications to the state. Depending on spedfic interests 
and resources, states can examine whether providers are meeting expectations related to: 

# Tutors' experience and gualifications 

# The amoimt of tutoring time students receive 

# The teaching strategies used 

# Instructional grouping and student- instructor ratios 

# Communication with teachers and parents 

# Promised transportation of students to and from tutoring 

In our prototype evaluation model (see section VI), we include such assessments in the Compliance 
category. Depending on the severity of compliance violations, providers may be downgraded from the 
highest status rating (e.g.. Full Standing) to a lower, probationary status, or in extreme cases, removal. 





6 Evaluating SES Providers 



Basic Components of a Comprehensive Evaluation 

* Ctommimicate evaluation plan to all stakeholders, including providers. See Sec- 
tion VII. Cornmunicating Evaluation Plan and Results. 

* Collect customer satisfaction, compliance, and achievement data during the 
school year. See Section III. Determining Evaluation Measures. 

* Synthesize data from multiple sources to rate the provider's performance on key 
categories (e.g. Student Achievement, Communications, Instructional Plans, 
etc.). See Section VI. Provider Evaluation Rubric and Decision Tree. 

* Obtain interrater corroboration on ratings, particularly those indicative of unsat- 
isfactory performance. See Section VI. Provider Evaluation Rubric and Decision 
Tree. 

* Use ratings in different categories and a formal "decision" process to judge pro- 
vide overall status for the next year. See Section VI. Provider Evaluation Rubric 
and Decision Tree. 

* Communicate results in appropriate manner to the state, districts, providers, 
and the public. See Section VII. Communicating Evaluation Plan and Results. 



III. DETERMINING EVAEUATION MEASURES 



This section outlines possible assessment measures for a comprehensive evaluation and monitoring 
effort. Both technical and practical considerations are offered for each measure to help states select an 
evaluation design that meets their needs. Suggestions on howto incorporate these measures into an 
evaluation design are provided in Section IV. Evaluation Designs: Student Achievement. 

A. Effectiveness Measures 

Measures of impact on student academic achievement are critical to a state's evaluation of SES provid- 
ers. This is especially true because NCLB reguires that, at a minimum, states remove providers from 
their approved list if the provider fahs to increase students' achievement for 2 consecutive years. State 
evaluations could measure achievement levels through state- mandated assessments, supplementary 
individualized assessments, or provider- developed assessments. 

1. Student- Level Test Scores From State-Mandated Assessments 

Evaluations of SES providers can examine students' scores on yearly achievement assessments that are 
administered by the state in compliance with NCLB. 

Technical considerations: 

* Such data may be available only for certain grades (e.g. Grades 3 to 8). Thus, some students 
who receive SES may be excluded. 

* Without students' pretest scores (e.g, those from the previous year), the gains associated 
with SES cannot be determined. 
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Why Are Pretest Scores Important? 



Example A: A third- grade student vdio received SES tutoring during the school year 
scores in the 60th percentile (above the national norm) on the spring state testing. An 
initial impression is to view the SES provider as having contributed positively to this 
relatively high performance. Although this might be the case, the child was not tested as 
a second-grade student the prior year. Perhaps, if she had been tested in second grade, 
she would have performed comparably or even relatively higher (e.g., the 65th percen- 
tile). In that case, the evaluator's interpretation would be guite different. 

Example B: Using a treatment control group evaluation design, the evaluator compares 
125 second-grade students who received SES from Totally Terrific Tutoring, Inc. (T3) 
to 328 SES-eligible second-grade students from the same district who did not elect to 
receive tutoring. The comparison group significantly outperforms the SES group on the 
state- mandated achievement test. T3 is very disappointed with this finding. However, 
had the second grade students been pretested (prior to SES tutoring), the evaluator 
would have found that the SES students were much more disadvantaged and lower per- 
forming than the control group. Was the T3 tutoring actually ineffective? 



# Achievement gains based on test scores at the secondary level (e.g., using benchmark, end- 
of- course, or gateway exams) may be difficult to analyze because students may take tests at 
diff erent times, in diff erent grades, and multiple times. 



* Although a comparison evaluation design is usually desirable, an appropriate group of 
control students is needed. Students in the control group should be similar to students 
enrolled in SES in demographic and prior achievement variables, and th^r must not have 
received SES. Section IV. Evaluation Designs: Student Achievement includes more details 
about this type of evaluation design. If data from a suitably large sample of SES and non- 
SES are available, it may be feasible to create a comparison group by adjusting statistically 
for individual and group differences in ability and background in the analyses of achieve- 
ment scores (see below). 



Practical considerations: 

# Identifying and collecting data on control students may be expensive and time- consuming. 

As discussed in section IV, a more practical (but probably less powerful) evaluation design 
is a multiple regression- type study that collects data on all students (both SES and non- SES) 
in a given subpopulation, such as those enrolled in Title I schools in the districts concerned. 
The analysis determines whether SES participation is associated with higher- than- expected 
achievement scores after controlling for student demographics (e.g., gender and ethnicity) 
and prior achievement scores. 

# In most cases, states may not have sufficient time to analyze the state assessment data before 
providers are approved to offer services for the next school year. For example, if students take 
the exams in the spring, then states may not receive the individual student score results until 
mid- summer. This leaves little time to analyze the data and evaluate providers before school 
begins. 

# To permit rigorous analyses, students' actual scores (e.g., number correct, scale score, or 
standard score) on the state assessments are needed. Merely knowing their proficiency levels. 
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such as "advanced" or "below basic," usually will not provide the degree of precision needed 
to detect differential growth for SES versus non- SES students. 

2. Supplementary Individualized Assessments in Reading/Language Arts or Math 

States can also use student scores on other valid assessment measures (e.g., an individually adminis- 
tered reading inventory, such as the Woodcock-Johnson test battery or the Durrell Oral Reading test). 
Such tests can be administered to students enrolled in SES (and possibly comparison students) to assess 
achievement in areas in which state-mandated test data are not available (e.g., for first-grade students), 
or to obtain more in-depth or rigorous measurement of student achievement. 

Technical considerations: 

# As mentioned previously, without pretest scores, the gain or benefit associated with SES can- 
not be determined. 

# Without comparison students, there will be no basis for judging whether SES students are 
scoring higher than peers vdio did not receive the tutoring services. For example, high scores 
from SES students could mostly be a reflection of the state's improved reading standards and 
curriculum, or some other systemic reform. 

# Unless these assessments are administered by trained independent testers, their validity may 
be suspect. Specifically, classroom teachers may lack adequate training or be imable to follow 
the protocols (e.g., completion time, room conations, etc.) required for administering tests 
accurately and consistently across students. SES tutors, obviously, would be put in an awk- 
ward position if they were responsible for administering the very tests from which they and 
their programs would be judged. 

Practical considerations: 

# Administering supplementary tests can be very time-consuming and expensive for states. 
Specifically, the evaluator will need to obtain copies of the test, locate students receiving SES 
(and presumably comparison students), and administer and score the tests at least once. 

# States using supplementary assessments may need to ensure that the timing of these tests 
does not conflict with state assessments or take additional time away from instruction. 

# States will need to determine whether the supplementary tests will be administered at the 
school site (e.g., during regular classroom hours) or on the SES provider's site and identify 
vdiose help will be needed to administer the tests. Note that if the tests are administered at 
the provider's site, equivalent testing for comparison students would be precluded. 

3. Provider-Developed Assessments in Reading/Language Arts or Math 

Some providers have developed or implemented their own assessment instruments to measure student 
progress. Many providers have included descriptions of these assessments in their applications to the 
state. Use of such diagnostic and formative tests is clearly a positive program component and should 
be stron^y encouraged. However, the question of whether the results from the providers' assessments 
should be a formal component of the states' evaluation of provider effectiveness raises critical technical 
and practical considerations. 

Technical considerations: 

# A provider's assessment may work well for its own informational purposes but may not pro- 
duce results that are valid or suitable for a state's evaluation purposes. That is, high perfor- 
mance on a provider's test may not translate necessarily into proficient performance on the 
state's test. 

# If providers use different assessments, then the state will not be able to compare performance 
across providers. 
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# Assessments that are specifically tailored for a provider's program may be more likely to show 
increases in student performance than would an externally developed test. 

# As noted above, the objectivity and validity of scores may be compromised when the provid- 
ers themselves are asked to administer and score tests that will be used to judge their effec- 
tiveness. For many providers, what had been viewed as highly valuable diagnostic and forma- 
tive tests would now become "high stakes" assessments with significant implications for their 
status and reputation. 

Practical considerations: 

# This option could save a state some time and money because it would not have to administer 
an assessment specifically for SES. However, sifting through results from a number of differ- 
ent assessments could be confusing and time-consuming. 

# For more informal or "formative" evaluation purposes, states could encourage or reguire pro- 
viders to use the same assessment instruments to measure SES students' progress. Because 
the providers would be administering those tests, the data may not be sufficiently indepen- 
dent of bias, as discussed previously. However, districts and educators may find consistent, 
formative data helpful forjudging individual student progress, and states may find such data 
useful for obtaining additional, informal impressions of a provider's impact. 

B. Customer Satisfaction Measures 

Parents, families, and students are SES providers' most important customers. To collect information on 
customer satisfaction, states may interview or survey students or parents of students enrolled in SES 
programs. This section identifies technical and practical considerations for including parent and stu- 
dent perspectives in an evaluation of SES providers. 

1. Parent and Family Perceptions 

Perspectives from parents or caretakers can play an important role in developing a complete picture of a 
provider's impact. By law, parents choose a SES provider for their children and can move their children 
to another provider if they are dissatisfied with the guality of services offered. When selecting a provider 
for their children, parents may want to consider what other fa mili es think about that provider. To help 
parents make informed choices, states could publish summaries of each provider's customer satisfac- 
tion results (see section VII). A sample survey that CREP/EI has used in several state evaluations is 
provided in Appendix A. Sample Instruments/Tools. 

Technical considerations: 

# Parents and fa mili es may be fa mili ar with only one SES provider, and may they not be able to 
compare that provider's services with the services of other providers. For that matter, except 
for the rare situations where SES tutoring occurs in the home, parents may have little or no 
impression of the guality or nature of the services provided. 

# The number of students that each provider serves can vary significantly; thus, the representa- 
tive sample size of parents will also vary. For example, if only a small number of students re- 
ceive services from a particular provider, the impressions of their parents may not accurately 
characterize the provider's services on average. 

# Parents vdio participate in surveys and interviews may not represent the population of par- 
ents vdiose children the provider serves (see next section). 

Practical considerations: 

# Identifying, contacting, and soliciting participation by the SES- involved parents may require 
extensive time and effort. 

# In the interest of protecting the confidentiality of SES participants, conducting interviews 
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would generally be precluded. Even if approval was obtained, interviews would be logistically 
challenging for the evaluators. Many parents do not like strangers contacting them by phone 
to ask guestions about their child. Probably, even fewer would want to take the time to attend 
an in-person interview or have an interviewer show up at their home. 

# Paper-and- pencil (not electronic) guestionnaires seem the most practical way of obtaining 
parent reactions. The problem with electronic surveys is that many parents (in this predomi- 
nately low- socioeconomic status population) may not have computers or be computer liter- 
ate. Due to confidentiality issues, the evaluators may not be approved to mail or distribute the 
surveys to the parents. Therefore, we have found that the most viable procedure is mailing 
the surveys to a school liaison who then mails or sends them via the students' "homework 
packets" to the parents. The parents, in turn, return the surveys to the school liaison who 
then mails them to the evaluators. 

# States may have to use translators to communicate with parents who do not speak En^ish. 

2. Student Perceptions 

Students enrolled in SES are the provider's primary customer and may be able to offer important feed- 
back about the guality of a provider's services. Gathering and reporting student feedback on providers 
may help states gain a more complete picture of a provider's impact and may also help parents make 
informed choices. However, as conveyed below, we have serious reservations about the viability of sur- 
veying/ interviewing students from both a logistical and validity (accuracy) standpoint. 

Technical considerations: 

# Students, especially younger children, may have difficulty judging the guality of services and 
communicating their impressions. 

# This type of data can be time-consuming to obtain, especially given the strong likelihood that 
evaluators will need parents' permission to survey students. 

Practical considerations: 

# Identifying, contacting, and soliciting participation of students enrolled in SES may reguire 
significant time and effort. 

# If confidentiality reguirements prohibit the evaluators from knowing the identity of SES 
students, guestionnaires would need to be administered for the evaluator by school or district 
personnel. This would significantly increase the logistical demands for administering gues- 
tionnaires while precluding interviews. It seems almost certain that sample sizes will be small 
and nonrepresentative of the population receiving the individual provider's services. 

# States may have to use translators for interviews or guestionnaires with ELL students. 

C. Service Delivery and Compliance Measures 

States may include service delivery measures in their SES provider evaluation.^ Questions about service 
delivery may include: Did the provider deliver the services it promised with regard to (a) the experience 
and gualifications of instructors; (b) the amount of tutoring time received by students; (c) instructional 
strategies used; and (d) its communication with schools, districts, and parents about student progress? 
Did the provider meet its contractual and legal obligations with the school district? Service delivery 
measures also address whether a provider complies with applicable laws and contractual procedures as- 
sociated with SES, including federal, state, and local health, safety, and dvrl rights reguirements. 



'^For more details on this measure, states should check district contracts with providers and ED's nonregulatory guidance on 
SES (www.ed.gov/ policy/ elsec/ guid/ suppsvcsguid.pdf) . States may also consider incorporating into their evaluation elements 
of the Education Industry Association's Code of Professional Conduct for Supplemental Educational Service Providers (www. 
educationindustry.org) . 
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To measure service delivery, states can review official records and obtain feedback from SES customers 
and district and school staff. 

1. Records of Services Provided, Student Attendance Rates, and Costs 

Evaluation studies cannot be performed meaningfully without a clear understanding of the exact inter- 
vention or "treatment." State or district records will help demonstrate vdiich services the provider actu- 
ally delivered to students (e.g., the number of students served, the average number of tutoring sessions 
provided, and the mean length of each tutoring session). States should also track student attendance at 
SES sessions to accurately judge a provider's performance. Suppose, for example, that a particular SES 
provider failed to show significant increases in student achievement. Examining attendance records 
may indicate that few students regularly attended their tutoring sessions. Therefore, the provider's lack 
of success in raising achievement may merit further investigation: Is the provider making a sufficient 
effort to implement a SES program that engages students' interests? Is the provider offering services at 
a time or in a place that is inconvenient for students or parents? 

Tracking student attendance and total hours of tutoring received is also important for increasing the 
rigor of analyses of provider effectiveness in raising student achievement (see Section IV. Evaluation 
Designs: Student Achievement) . That is, including in such analyses students vdio have received only a 
few hours of tutoring would bias the results. But what is "minimal" or "adeguate" tutoring? States need 
to define such levels. 

Technical considerations: 

# To obtain accurate information about service delivery, states may obtain data from a variety 
of sources, including providers, teachers, principals, and district staff. States may choose to 
obtain data during announced or unannounced site visits. During these visits, which may be 
part of routine monitoring by state staff, the observers can view providers delivering services 
and speak with the provider, students, families, teachers, principals, and district staff. Cor- 
roborating data from multiple sources can increase the accuracy of evaluation conclusions. 

Practical considerations: 

# Data may not be readily available from schools and districts. States should establish clear 
procedures that require districts and providers to maintain SES attendance records. Using 
an automated, centrally controlled (ty states), Web- based data recording system is stron^y 
recommended. Several commercial vendors offer services and software for these purposes. 

# State employees (or outside contractors) must be dedicated to collecting and analyzing data. 

# Onsite visits, if desired by the state, will demand significant time, labor, and resources. 

2. Feedback From SES Customers 

Measures of parent and student experiences and perceptions not only offer information about a provid- 
er's effectiveness, but they can also reveal details about a provider's service delivery. 

Technical considerations: 

# As previously discussed, parents may lack firsthand impressions or observations of the ser- 
vices being delivered. 



Practical considerations: 

# Obtaining representative samples of respondents may be difficult given mobility, availability 
of data, and willingness of prospective respondents to participate. Still, unless the sample is 
obviously nonrepresentative, too small, or biased, the impressions provided should be useful 
as part of the overall evaluation. 
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# States may have to use translators to reach parents who do not speak En^ish. 

3. Feedback From District Staff 

Local school districts are a critical component of effective SES implementation. Districts contract with 
and pay providers and are often in closer contact with the providers than the state. As a result, districts 
may be able to offer crucial feedback about a provider's service delivery. Online surveys, similar to 
samples in Appendix A. Sample Instruments/ Tools, are used for SES coordinators in each district with 
schools required to offer SES. 

Technical considerations: 

# District administrators may lack firsthand impressions or observations of tutoring services. 
Practical considerations: 

# Some districts may also be SES providers. Thus, a district may not be able to provide an 
unbiased review of its own services and may not be comfortable or able to provide an honest 
review of its competitors' services. 



4. Feedback From Schooi Staff 

Principals and classroom teachers have firsthand knovdedge about a student's in- school performance. 
They can also offer valuable input about a provider's services. In particular, teachers of students en- 
rolled in SES can offer feedback about the impact of the provider's services on a student's progress, and 
they can determine whether and to what extent a SES provider works with school staff to assess the 
student's needs, aligns instruction with the district's instructional program, and reports back to school 
staff about the student's progress. A sample survey instrument (CREP, 2006) is shown in Appendix A. 
Sample Instruments/ Tools. 

Technical considerations: 

# Teachers may lack firsthand impressions or observations of providers' service delivery. 

# Teachers may also be SES instructors. In such cases, a teacher's dual role should be explicitly 
noted so that responses can be interpreted in view of possible bias. 

Practical considerations: 

# Identifying, contacting, and soliciting participation by the SES- involved teachers may require 
time and effort for the school. Typically, the evaluator will not know the identities of the 
teachers involved, and will mail the surveys to the school for distribution internally. 

# Teachers may need to provide information about multiple providers, which may be confusing 
and time-consuming, leading to fewer surveys being completed. 

Data Sources 

Once states have chosen specific evaluation outcomes, they will need to design a research plan for col- 
lecting and analyzing the applicable data. Table 1 summarizes possible sources that states can use to 
obtain the data needed to address their respective evaluation questions. 
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Table 1. Summary of Data Sources by Question 



Evaluation question 


Data source I 


1 . Does the SES program raise student achievement 
in reading/language arts and/or math? 


• Pretest and posttest achievement data in 
reading/language arts and math 


2. Does the SES provider work with principals, teach- 
ers, and parents as needed to develop instruc- 
tional plans that are geared to the needs of the 
students and aligned with state standards? 


• Principal/liaison survey 

• District coordinator survey 

• Teacher survey 

• Parent survey 

• Provider survey 


3. Does the SES provider communicate effectively 
with principals, teachers, and parents about the 
progress of students enrolled in SES? 


• Principal/liaison survey 

• District coordinator survey 

• Teacher survey 

• Parent survey 

• Provider survey 


4. What are teacher, student, and parent experiences 
with and reactions to SES interventions? 


• Teacher survey 

• Parent survey 

• Student survey 


5. Does the SES provider meet the needs of ELL 
students and students with disabilities? 


• District coordinator survey 

• Principal/liaison survey 

• Teacher survey 

• Provider survey 


6. Does the SES provider deliver the services it 
promised in its application and/or contract with 
regard to (a) the experience and qualifications of 
instructors, (b) the amount of tutoring time stu- 
dents receive, (c) instructional strategies used, 
and (d) its communication with schools, districts, 
and parents about student progress? 


• District coordinator survey 

• Principal/liaison survey 

• Teacher survey 

• Provider survey 

• Parent survey 

• Monitoring documentation 


7. Does the SES provider meet its legal and contrac- 
tual obligations? 


• District coordinator survey 

• Principal/liaison survey 

• Teacher survey 

• Provider survey 

• Parent survey 

• Monitoring documentation 




















14 Evaluating SES Providers 



Technology and Database Considerations 

States will need to collect a large amount of data to evaluate SES providers. To effectively and 
efficiently collect data, states may consider developing and maintaining a relational database 
to store information that connects: 

1. Achievement data and related characteristics (including demographic information) 
for all students vdio are eligible for SES 

2. Each student SES serves with a specific SES provider 

3. Details about the services offered each SES provider offers 

The database serves as a central repository for data about SES and facilitates student- level 
achievement analyses. By keeping track of detailed information about providers and students, 
the database can be used to conduct in-depth analysis on the impact of specific provider prac- 
tices, such as instructional approaches, student grouping, or service duration. Analysts can 
also look at how a provider's impact varies by student characteristics (e.g., grade level, race, 
gender, socioeconomic status, and location). 

Assuming that not all eligible students participate in SES, maintaining information about 
"eligible but not participating" students could lay the groundwork for a possible quasi- experi- 
mental (e.g., matched samples), student- level design. The information in the database could 
also be aggregated to provide the foundation for subsequent analyses at the school, district, 
and provider levels. 

School or district SES coordinators will likely need to gather and submit data. Data gather- 
ing and submission could be accomplished through a Web- based data entry mechanism or 
through forms that are distributed to the local points of contact and returned to the state for 
entry into the database. The data entry could be facilitated through standardized procedures, 
including a user-friendly form and predetermined submission deadlines (e.g., quarterly da- 
tabase updates). Here is a sample of a student information table. (Additional fields, such as 
those for demographic categories, would likely be needed, as This is a generic example.) 



Sample Student Table 



LEA 


School name 


School ID 


Student last 
name 


Student first 
name 


student ID 


Grade 


Receiving 

services 

(Y/N) 


Provider 

name 


SES 

Subject 

area 
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IV EVALUATION DESIGNS: STUDENT ACHIEVEMENT 



After identifying evaluation outcomes and potential data sources, states can begin to select the evalua- 
tion instnunent(s) that will enable them to conduct an effective evaluation process. At minimum, NCLB 
reguires that states assess student achievement to determine the provider's effectiveness in the tutored 
subj ects (reading/ language arts and/ or mathematics) . States should note that guantitative results rely 
greatly on the scientific rigor of the assessment's design; that is, the more rigorous the design, the more 
confidence can be placed in the results. Because each evaluation option reguires the state to assemble 
comprehensive datasets and a guantitative research team dedicated to the analysis, states will also need 
to carefully weigh the technical merits of each design against practical considerations, such as avail- 
able resources and time. For more details on maintaining data, please see the Technology and Database 
Considerations box on page 15. 

This section outlines possible designs that states can use for SES evaluations. Each option has been 
rated from "+” (lowest scientific rigor) to " MM " (highest scientific rigor). Technical and practical 
considerations are also included to help states select the design that best fits their chosen evaluation 
outcomes and data sources. However, in describing different options, we want to stron^y emphasize 
the importance of states making every effort to implement a design that is rigorous ("-H-I-" or "-I-I-I-I-"). 
A basic tenet of educational research is that weak evidence can be more harmful than having no evi- 
dence at all. Think of the conseguences to children, schools, and the provider industry of erroneously 
identifying individual providers as effective or ineffective. Newspapers and other media are often guick 
to provide exaggerated characterizations of outcomes without explaining the nuances, as reflected in 
a headline such as "SES Tutoring Fails to Raise Student Achievement." Quite possibly, the evaluator 
simply compared SES and non- SES students on the spring state assessment, without adjusting for the 
higher poverty and lower prior achievement of the students enrolled in SES. A more rigorous evaluation 
might have demonstrated that SES tutoring was highly effective at fostering year-to-year gain. 



A. Benchmark Comparison 

In this evaluation design, the aggregate percentage of students attaining "proficiency" on the state as- 
sessment is computed for each provider and descriptively compared to established benchmarks set by 
the state. Confidence intervals can be constructed aroimd each provider estimate to determine statis- 
tical reliability. In general, this design is not sufficiently rigorous for makingjudgments of provider 
effectiveness that would result in major status decisions (e.g, removal of a provider from a state- ap- 
proved listing) . However, where resources or time are limited, it could provide informative "suggestive" 
impressions about tutoring outcomes. A more rigorous design might then be implemented the following 
year. 

Example: Provider A has provided math tutoring to 254 students who have test scores in mathematics 
on the state assessment. Of these students, 82% score at the "proficient" level or higher, which exceeds 
the state benchmark for the year of 76%. Computation of a confidence interval aroimd the estimate sup- 
ports the conclusion of a statistically significant advantage for the Provider A students. 

Rating = -i- (low rigor) 

Advantages: 

# Relatively inexpensive from a data processing and analysis perspective. 

* Easily understood by practitioners and the public. 
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# The same dear standard, directly linked to NCLB accountability, is applied to all providers. 

Disadvantages: 

# Confounding of provider and school effects. For example. Provider A may have a large pro- 
portion of its students clustered in a small number of schools that are adopting new research- 
based curricula. 

# Does not capture student growth along a score continuum (e.g, 0 to 100). That is, perfor- 
mance categories such as "profident" or "not profident" can indude a wide range of scores, 
therefore sacrifidngpredsion of measurement. 

# Does not take into account variability in outcomes that may be strongly related to individual 
student charaderistics. For example. Provider A might be serving a higher risk student popu- 
lation (lower achieving, ELL, and spedal needs students, etc.) than other providers. 

# May create adverse incentives; that is, providers avoid serving students who are most at-risk. 
To the extent that "profidency" is established as the criterion for success, moving students 
from very low 'below basic" levels to higher (but still below profident) levels will not be con- 
sidered. 



Rating = -f-f (low to moderate rigor) 

Strategies for improving this design wouid inciude: 

# Including all performance categories (e.g, below basic, basic, profident, or advanced) rather 
than only the percentage meeting benchmark 

# Determining SES students' improvement in performance relative to the prior year and com- 
paring the improvement (or gain) score to that for the state overall 

# Comparing yearly performance or gain of SES students to a control sample of similar non- 
SES students 

B. Multiple Linear Regression Design (Using Pretest and Posttest) 

Using a statewide database, a Multiple Linear Regression (MLR) model is estimated in vdiich the cur- 
rent year's achievement score is predicted by prior achievement, free or reduced- price lunch status, 
and other pertinent predidor variables. A "fixed effect" (effectiveness score) for each provider can be 
estimated and subjected to an inferential statistical test. This test would determine whether the effect is 
statistically reliable rather than likely to occur by chance. 

More simply, this design uses correlation- type analyses to generate estimates of a student's future test 
scores based on his/ her previous test scores and other student demographic variables. If students en- 
rolled in SES actually perform higher than the estimates and if students not enrolled in SES perform at 
or lower than the estimates, then researchers can reasonably infer that the SES provider shows a posi- 
tive effect on student achievement. Conversely, if students enrolled in SES score at or lower than their 
predicted scores, researchers can conclude that the provider shows no or even a negative effect. 

Example: There are 45 fourth- grade students who receive reading tutoring from Provider B. Based on 
the students' third- grade reading scores, gender, and ethnicity, the MLR model estimates how each is 
likely to perform on the fourth- grade assessment. Students' actual scores are then compared to their 
predicted scores. Overall findings indicated that the average student scored 10 points higher than pre- 
dicted! The 10 points reflect a one- half standard deviation advantage, which translates into an effect 
size of -K).50 (very strong). Further, the effect size is determined to be statistically reliable, with less 
than a 1% probability of occurring by chance. The provider therefore is evaluated as "effective" in raising 
student achievement. 
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Rating = +++ (moderate rigor) 

Advantages: 

# Although more expensive than the Benchmark Comparison, the MLR approach is still rela- 
tively economical with regard to time and cost. That is, students enrolled in SES do not need 
to be individually matched to similar control students (see design C, Matched Samples De- 
sign). 

# By taking student characteristics into account, a more precise determination of the provider's 
unigue contribution ("effects") can be determined. Note that because prior test scores are the 
strongest correlate of subseguent achievement, the absence of pretest data will reduce the rig- 
or of the MLR (and other design options) considerably. In fact, we recommend that definitive 
judgments about provider performance not be made on the basis of posttest-only designs. 

# The effect size estimates (see above example) can be combined across grades and years via 
meta- analytic procedures to increase power to detect small provider effects over time. 

Disadvantages: 

# Confounding of provider and school effects. For example. Provider A may have a large pro- 
portion of its students clustered in a small number of schools that are adopting new research- 
based curricula. 

# Less easily understood by practitioners and the public than the Benchmark Comparison. 

# The achievement estimates may be less stable and efficient than those derived from more 
sophisticated approaches (see design C, Matched Samples Design). 

C. Matched Samples Design 

There are a variety of approaches to creating matched samples. The simplest approach involves match- 
ing each SES student with a similar student attending the same school. Provider effects are then com- 
puted relative to this comparison sample using inferential statistical analyses. 

Example: Provider C has tutored 78 students in reading across the state. For each of the 78 students, 
the evaluator identifies a matched control student counterpart vriio (a) attends the same school or, if not 
possible, a similar school in the same district; (b) is the same in gender, poverty (free or reduced- price 
lunch status), and ethnicity; and (c) scored identically or very similarly in reading on last year's state 
assessment. Reading scores on this year's state test are then compared for the matched- pair SES and 
control groups. 



Rating = -f-i-i-f (high moderate to strong rigor) 

Note that greater precision and efficiency can be achieved through use of multiple matches per SES 
student. This procedure might involve matching SES students to highly similar students within their 
school and to those attending other similar schools. Thus, for example, if SES student #306 matched 
well to five non- SES students at her school and two neighboring schools, it would increase statistical 
power and precision to use the multiple matches rather than only one. Another means of increasing 
precision is using more sophisticated statistical procedures that match students via "propensity score 
weights." 

Advantages: 

* Provides some control over school effects (where there are within- school matches) 

* Conceptually easy to understand and has strong "face validity" 
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# Pennits computation of traditional effect size estimates 

# Probably the most rigorous approach available 
Disadvantages: 

# Substantially more time- consuming than the Benchmark Comparison or MLR approaches 
from a data analysis perspective. 

# In some cases, there may be too few appropriate within- school matches, thus reducing ability 
to control for school effects. This problem can be ameliorated by using more complex match- 
ing strategies, but the straightforward appeal of simple within- school matching would be 
compromised. 



V EVALUATION DESIGNS: CUSTOMER SATISEACTION AND SERVICE DELIVERY 



Table 2 summaiizes possible sampling strategies and data collection tools^ that states can use to assess 
a SES provider's customer satisfaction and service delivery outcomes. In general. Table 2 suggests that 
surveys or interviews from at least five parent and teacher respondents per provider should be obtained 
vdien possible. A larger and more representative sample will result in greater confidence in the validity 
of resiilts. In many instances, evaluators will need to decide subjectively and defend the level of confi- 
dence that can be attached to the data. In the case of parent surveys in particular, obtaining "represen- 
tative" samples will be very challenging or not even possible. Specifically, respondents are likely to be 
disproportionately those who (a) are more involved with the children's education, (b) are more literate 
or educated, or (c) have particular positive or negative feelings or personal agendas regarding the tutor- 
ing services. 

How many respondents are enough? The number of students served may vary considerably across 
providers within a state. One option is to establish target sample sizes based on a percentage of students 
served. However, the result may be imreasonably large or small numbers for particular providers. We 
prefer a pragmatic approach that strives to include as many survey respondents as feasible in order to 
increase sample size. Below are two scenarios in which results from small samples might be interpreted 
differently by the evaluator. 

Example 1: Provider A serves 32 children in a school district. Of the corresponding 32 surveys sent 
home to parents, only 8 are returned. However, all 8 show consistently positive ratings, while including 
open-ended comments noting specific beneficial activities of the provider (e.g, calling the parent, relat- 
ing well to the student, and increasing motivation and learning) . These impressions corroborate those 
from other sources. Although sample size is small, the evaluator feels confident in interpreting these 
results as highly supportive. 

Example 2: Same as example 1, but the eight returned surveys reveal highly varied ratings and open- 
ended comments focusing on general and personal interests ( 'Tutoring is good for my child. " 'TTie tutor 
seemed nice."). These results and the small sample size lead the evaluator to regard the evidence as 
insufficient or inconclusive at this time. 



^CREP has developed scannable and online surveys. For more information, go to http:/ / crep.memphis.edu. Also, samples of 
selected surveys may be seen in Appendix A. Sample Instruments/ Tools. 
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Table 2. Sampling Strategies and Data Collection Tools to Assess Customer Satisfaction and 
Service Delivery 



Respondent 

group 


Sampling strategy 


Data collection tool 


District SES coordinators 


Usually one coordinator per district; if 
more, the evaluator may want to include 
all. 


Survey of district coordinators 
with follow-up phone interview if 
needed 


Principals or school 
representatives 


At least one representative at each SES 
school, responding for each provider 
serving students at that school 


Survey of principals or school 
representatives with follow-up 
phone interview if needed 


Teachers 


At least 8 but preferably as many as 
15-20 teachers per provider, depending 
on the number of students served 


Survey of teachers 


Parents 


At least 5 but up to 20 per provider, 
depending on the number of students 
served 


Survey of parents 


Providers 


One representative from each provider 


Survey of providers 


Students (when feasible) 


From 5-1 0 students per provider, 
depending on the number of students 
served 


Survey of students 



VI. PROVIDER EVALUATION RUBRIC AND DECISION TREE 



A state's overall evaluation of a SES provider should link all three dimensions of performance: 

L Effectiveness. Measured by changes in student achievement in reading/ language arts or math. 

2. Customer satisfaction. Measured through surveys or interviews with students and parents of 
students who receive SES. 

3. Service delivery and compliance. Measured through surveys or interviews with principals, 
teachers, and school district staff to determine compliance with applicable laws and contractual pro- 
cedures associated with the delivery of SES. 

States will need to synthesize data from multiple sources vdien conducting a summative evaluation of 
each provider. Based on the summative evaluation results, each provider's status for the following year 
can be determined. Figure 1 provides a graphic representation of a summative evaluation.® 

To facilitate this process, states may choose to develop a culminating tool, or an "overall provider evalu- 
ation rubric," to synthesize the multiple data sources that reflect providers' activities and outcomes. 

This tool can help states clarify how well each provider meets expected outcomes and standards. The ru- 
bric can be organized by the major assessment categories (e.g., student achievement gains and effective 
communications) that are included in a state's SES evaluation plan, and each category should be de- 
fined. For each category, a scoring rubric— consisting of four levels of attainment (e.g., above standards, 
acceptable, marginally acceptable, and below standards) and accompanying written definitions— can be 



®This model is also used in CREP's comprehensive SES Evaluation/ Monitoring Plan (see http:/ / crep.memphis.edu) . 
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constructed. For example, an "above standards" level for student achievement gains and for effective 
communications may appear as follows: 

# Student achievement gains. SES students' average gain shows an effect size of +0.XX^ or 
greater, relative to the average gain for control students or as indicated in appropriate norms. 

# Effective communications. Based on interviews, all of the district's coordinators, school 
principals, liaisons, teachers, and parents rate the provider positively in communicating SES 
activities, student needs, and progress. The provider also produces concrete evidence sup- 
porting the use of effective communications with these groups. 

Figure 1 . Summative Evaluation Overview 



Figure 1. Components of a Comprehensive SES/Evaluation Modeling Plan 



Service 

Delivery 

Provider 

Survey 
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Customer 
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I 



STUDENT 

ACHIEVEMENT 



state 

Tests 



I 



Additional 



Overall Provider Assessment 



Rubric Application 

A sample rubric used in a state evaluation conducted by CREP may be viewed in appendix B.^ In addi- 
tion to evaluating student achievement, this particular state established supplemental core categories, 
including: Communication, Instructional Plans, Local and State Standards, Special ED/ ELL Students, 
and Overall Performance. For each category, the evaluator synthesizes the available data to select one of 
five ratings: 

# Insufficient information 

# Below standards 

# Marginal guality 

# Acceptable 

# Above standards 



'States will need to determine the effect size. 

“CREP has developed an electronic template and scoring system that automatically records the ratings and stores them in a 
database for further analyses and possible comparison to state or national norms (depending on SEA preference) . 
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To ensure reliability of findings, each evaluation is reviewed by at least two independent raters. Where 
there is disagreement, a consensual rating is derived. 

Decision Tree Application 

Once the rubric ratings have been completed, the culminating step is determining each provider's sta- 
tus for the following year. Appendix C illustrates an example of a generic decision tree that CREP has 
devised for such purposes, with status category descriptions illustrated in appendix D. The assumption 
was that individual states could use the generic framework as a starting point but make adaptations to 
fit local policies or preferences. 

The decision tree establishes a hierarchy of status levels, ranging from "full standing" to "removal." 
Intermediate levels from highest to lowest include "satisfactory standing," "probation I," and "probation 
II." The decision logic is designed to give the strongest positive weight to providers that demonstrate 
"clear" effectiveness in raising achievement. Conversely, the strongest negative weight is assigned to 
dear failure to raise achievement but also to serious compliance violations.^ When student achieve- 
ment outcomes for a provider are indeterminate (which may be guite common, particularly in the first 
few years of data analysis, due to small sample size), the dedsion will assign more weight to imple- 
mentation guality (e.g., consumer satisfaction variables) and across years, to implementation improve- 
ment. This particular model, therefore, adopts a "developmental" orientation that attempts to foster 
aocoimtability and continuous improvement for providers that cannot yet demonstrate reliable effeds 
on achievement (mostly due to statistical or sampling constraints) but appear to be offering benefidal 
services. 

VII. COMMUNICATING EVALUATION PLAN AND RESULTS 



Two important areas of communication for states involve oommunicatingthe evaluation plan and 
expectations to providers and other stakeholders prior to the school year. In fact, states might consider 
reguiring providers to acknovdedge formally that they are both aware of evaluation reguirements and 
will cooperative with data colledion as needed. 

Once the evaluation is completed, states will need to communicate the results of provider evaluations to 
providers, families, schools, districts, and the general public. A state should consider whether and how 
it will report results in a user-friendly way. States will typically give more detailed information to pro- 
viders to help them understand strengths and weaknesses of their SES program and make needed im- 
provements. States may also choose to post summaries of each provider's evaluation results on the state 
Web site so that districts and fa mili es can access the information and use it to select a provider. To help 
ensure that information is provided in a central, accessible place, states can incorporate these results 
into their publicly available lists of approved SES providers. States should also consider whether they 
will share evaluation data with researchers who are interested in assessing the overall impact of SES. 



^Criteria for establishing "dear" (statistically significant and demonstrable) criteria for achievement effects and defining the 
seriousness of compliance violations would generally be dedded by individual states. Guidance or technical assistance is avail- 
able from the authors. 
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VIII. CONCLUSION 



The evaluation of SES providers is a relatively new and emerging endeavor, and evaluation models are 
still evolving. As is true in virtually all educational domains, no one evaluation approach is likely to fit 
all conditions and interests. Each state has unigue needs, priorities, resources, and procedures that it 
can use to implement SES. Therefore, this Guidebook presents a range of recommended outcome mea- 
sures and research designs for states to consider. The options presented focus strongly on the critical 
goal of assessing the effects of SES on student achievement. 

As the descriptions of the different evaluation options convey, states may face a tradeoff between practi- 
cal concerns (cost and time) and rigor (the reliability and accuracy of findings) . As a general approach, 
each state should begin its SES evaluation planning process by: 

# Identifying, through review of this Guidebook and discussions with key stakeholders, the 
specific guestions that its SES evaluation needs to answer. 

* Identifying the resources that can be allocated reasonably to support further evaluation plan- 
ning, data collection, analysis, reporting, and dissemination. 

With a clear understanding of its research needs and practical resources, a state can work through the 
hierarchy of evaluation designs presented here and select the design that allows the highest level of 
rigor. In some cases, states may adopt a hybrid approach in vdiich (a) a less rigorous but more practical 
design (e.g.. Regression Analysis) is broadly applied across the state, and (b) a highly rigorous design 
(e.g.. Matched Student Pairs Analysis) is used in selected contexts and then expanded each year. Evalu- 
ations may also include an analysis of providers' outcomes relative to the cost of their services. States 
may opt to engage third-party evaluation experts in helping to plan and conduct these evaluations. Such 
experts can lend experience, knovdedge, and credibility to the evaluation, if the evaluator is not biased 
toward a particular SES provider, district, or school. The What Works Clearinghouse's Registry of Out- 
come Evaluators is one source that states can use to find an evaluation expert. To search the registry, go 
to www.whatworks.ed.gov. 

Given the importance and complexity of evaluating SES providers, it makes little sense for each state 
to reinvent the wheel. Over time, SES evaluation experiences nationally will yield valuable information 
and improved data collection methodologies and tools. By exchanging tools and lessons learned with 
other states, officials can save considerable time and resources, but most importantly, they can improve 
the validity, practicality, and usefulness of their own SES evaluations. 
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IX. ADDITIONAL RESOURCES 



American Educational Research Association (AERA) 

www.aera.net 

AERA is an international, professional association of researchers that is focused on advancing 
educational research and its practical applications. 

American Evaluation Association (AEA) 

www.eval.org 

AEA is an international, professional association of evaluators. It is focused on improving the 
effectiveness of evaluation practices and methods that are used to assess the strengths and weaknesses 
of programs, policies, personnel, products, and organizations. 

Center for Research in Educational Policy (CREP) 

http:/ / crep.memphis.edu 

Based at the University of Memphis, CREP conducts research on educational policies and practices in 
pre-K- 12 public schools and provides evaluation instruments for educational practitioners. CREP's Dr. 
Steven Ross co-authored this issue brief with the SESQ Center. 

Center on Innovation & Improvement (CII) 

www.centerii.org 

QI is one of five content centers in ED's system of comprehensive centers, which also includes 16 
regional centers. CII's Web site provides information on SES and other areas of school improvement 
and innovation. 

Council of Chief State School Officers (CCSSO) 

www.ccsso.org 

CCSSO works with chief state school officials and their agencies to strengthen and support the 
American education system. CCSSO's Web site offers guidance and resources on afterschool programs 
and SES. 

National Reporting System (NRS) for Adult Education 

www.nrsweb.org 

NRS is an outcome- based reporting system for state- administered, federally funded, adult education 
programs. The Web site offers guidance on collecting, analyzing, and reporting the impact of adult 
education programs. 

Office of Innovation and Improvement 

www.ed.gov/ about/ offices/ list/ oii/ index.html?src=oc 

This office of ED coordinates public school choice and supplemental educational services programs 
along with the Office of Elementary and Secondary Education (www.ed.gov/ about/ offices/ list/ oese/ 
index.html). 

Practical Assessment, Research & Evaluation 

http:/ / pareonline.net/ 

This onlinejoumal provides education professionals with access to peer-reviewed articles on 
assessment, research, evaluation, and teaching practice, especially at the district level. 

Regional Educational Laboratories (REL) 

www.relnetwork.org 
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REL is a network of 10 educational laboratories that serve specific geographic regions of the United 
States. The laboratories provide information, research, and tools to help those involved in educational 
improvement at the local, state, and regional levels. 

Supplemental Educational Services Non-Regulatory Guidance 

www.ed.gov/ policy/ elsec/ guid/ suppsvcsguid.pdf 

This resource serves as ED's guidance on SES. It provides complete information for states and districts 
to implement provisions of SES. 

The Evaluation Exchange 

www.gse.harvard.edu/ hfrp/ eval.html 

This periodical from the Harvard Family Research Project highlights innovative methods and 
approaches to evaluation, emerging trends in evaluation practice, and practical applications of 
evaluation theory. Subscriptions are free, and it is published three to four times per year. 

User-Eriendly Handbook for Project Evaluation 

www.nsf.gov/ pubs/ 2002/ nsf02057/ start.htm 

This handbook from the National Science Foundation explains howto create and implement 
evaluations for educational programs. It offers information on guantitative and gualitative evaluation 
methods and guidance on locating external evaluators. 

What Works Clear in^ouse (WWC) 

www.whatworks.ed.gov 

WWC collects, screens, and identifies studies of the effectiveness of educational interventions. Online 
visitors can search the Clearinghouse's Registry of Outcome Evaluators to find an independent 
evaluator that may be able to assist with SES evaluations. 
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AMERICAN 

INSTITUTES 

FOR Research’ 






About AIR 

The American Institutes for Research® (AIR®) is one of America's leading educational 
research and evaluation organizations. Since its foimding in 1946 as an independent, not- 
for-profit corporation, AIR has provided school districts, states, and the federal government 
with consulting services in the areas of research, analysis, operations, technical assistance, 
assessment, and strategic communications. AIR is headguartered in Washington, DC, with 
10 regional offices throughout the United States and 12 international offices. AIR currently 
has over 1, 100 employees, more than half of vriiom work in the area of education, and many 
of whom are former school teachers, principals, and central office and state department of 
education employees. 

AIR staff have extensive expertise in evaluating the implementation and effectiveness of 
supplemental educational services (SES) and providing technical assistance to those vriio are 
responsible for SES, as well as expertise in engaging and empowering parents. From 2003- 
2005, air's Supplemental Educational Services Quality Center (SESQ Center— www.tutors- 
forkids.org), funded by the Office of Innovation and Improvement of the U.S. Department of 
Education (ED), supported effective implementation of SES. Through the SESQ Center, AIR 
provided research, technical assistance, products, presentations and consulting services to 
(a) engage families to build informed demand for SES, (b) increase the supply of high-guality 
SES providers, and (c) provide information and improved coordination among national orga- 
nizations that support the implementation of SES. 

Currently, AIR is assisting states with SES evaluation and outreach efforts through its work on 
several Regional Comprehensive Assistance Centers, funded by ED. In addition, AIR has part- 
nered with the Center for Research in Educational Policy to directly help several states evalu- 
ate SES providers. AIR is also working on the National Longitudinal Study of No Child Left 
Behind (NLS-NCLB), and the Study of State Implementation of Accountability and Teacher 
Quality Under NCLB (SSTNCLB). These linked studies are both 4- year, longitudinal evalu- 
ations. NLS-N CLB focuses on the implementation of NCLB at the district and school level, 
\Afiile SSTNCLB collects interview and extant data from states. Both studies have collected the 
most complete, nationally representative data that currently exists on the implementation of 
SES. 



Web site: www.air.org 



Phone: 202-403-5000 
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p 


/ 

f CREP 




'center for Research in Educational Policy 



About the Center for Research in Educational Policy 

The Center for Research in Educational Policy (CREP) is a University of Memphis- based re- 
search center and State of Tennessee Center of Excellence vdiose mission is to implement a 
research agenda associated with educational policies and practices in pre-K- 12 schools and to 
disseminate research findings so that they inform decisions made by educational practitioners 
and policy makers. CREP has gained national recognition for its contribution to discussions of 
issues such as reform of teacher education, educational ecjuity, educational technology, school 
reform and restructuring, urban and multicultural education, interventions for at-risk students, 
and using formative evaluation methods for school improvement decision making. In particular, 
CREP is one of the most active and expert organizations in guiding national policies and state 
practices for conducting SES evaluations. 

One major focus for CREP is to plan and conduct high-guality studies individually and in collab- 
oration with peer organizations to determine "vdiat works" in schools across the nation. In this 
regard, CREP has served as a consultant or direct contributor to research standards designed to 
address the goals of No Child Left Behind (NCLB), as proposed by the National Academy of Sd- 
enoes and the What Works Clearinghouse. 

In addition to this research focus, another priority for CREP is assisting schools in building their 
capacity and ability to make data-driven decisions for the purpose of attaining NCLB Adequate 
Yearly Progress standards. This emphasis was reflected duringthe 2005- 2006 school year in 
which approximately 2,000 schools across the nation used tools and strategies CREP developed 
as part of its Formative Evaluation Process for School Improvement (PEPSI). These tools ad- 
dress critical school improvement factors such as teaching methods and quality, school climate, 
teacher support, technology usage, program implementation and student achievement. Exem- 
plary projects in which CREP has been recently involved include: 

* Evaluator of Reading First in Tennessee and Louisiana 

* Evaluator of charter schools in Tennessee 

* Evaluator of educational technology initiatives in Tennessee, Michigan, and Kentucky 

* Evaluator of the New Leaders for New Schools Principal Development Program 

* Evaluator or Co- Evaluator of Comprehensive School Reform in Georgia, Tennessee, 
West Virginia, and Kentucl^?^ 

* Evaluator of SES in Tennessee, Louisiana, Virginia, Kentucky, Illinois, and Texas 

* Evaluator of Transition to Teaching Program in Tennessee 



Web site: www.crep.memphis.edu 



Phone: 866-670-6147 
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Center ON 

INNOVATION & IMPROVEMENT 

Twin paths to better schools 



About the Center on Innovation & Improvement 

The Center on Innovation &Improvement (QI) is one of five content centers in the U. S. 
Department of Education's system of comprehensive centers, which also includes 16 regional 
centers. QI's goal is to provide information, tools, and training to the regional comprehensive 
centers so they can help states improve student learning by: 

# Assisting districts and schools in their improvement 

# Implementing and supporting the choice provisions of No Child Left Behind 

# Increasing the number and guality of charter schools 

# Ensuring equitable participation of private school students and teachers in federal 
programs for vdiich they are eligible. 

The Academic Development Institute (Lincoln, Illinois) and its partners— Temple University 
Center for Research in Human Development and Education (Philadelphia, Pennsylvania) and 
Little Planet Learning (Nashville, Tennessee)— operate the CII with an experienced staff, a Sci- 
entific Council with distinguished scholars, a Leadership Council of experienced school admin- 
istrators, and technical advisors with specific expertise. 

To support the regional comprehensive centers' work with states to improve and expand 
supplemental educational services (SES), QI: (a) collaborates with the Center for Research in 
Educational Policy and the American Institutes for Research to provide up-to-date information 
on evaluation of SES services and systems, (b) assists regional centers in providing technical 
assistance on SES to state education departments, (c) provides training and technical assis- 
tance in a program model to engage community- based organizations and faith- based organi- 
zations in outreach to parents about SES, and (d) provides information about SES on its Web 
site, including research, reports, tools, state policies, state programs, and state progress. 

QI's other areas of priority are restructuring and state systems of support for schools in im- 
provement. Information about these priority areas and other QI projects may be found at QI's 
Web site. 



Web site: www.centerii.org 



Phone: 217- 732- 6462 
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Appendix A. Sample Instruments/Tools 



Supplemental Educational Services 
District Coordinator Questionnaire 

P'Mer. A, iScOmaltl. AJ.. 4- KW 
CMerJb* fOotofch in foiiey, Tfia Likiveriily i^Metupkit 

jiiJ /ogktj fUierMl 



District Name: O Davidson County O Hamilton County 0 Knox County 

O Fayette County -.I- Hardeman County Q Memphis City 

Name of Person 

Completing this Survey 

Title of Person 

Completing this Survey 



DIRECnONS 

-f mew iigWLOirY M 
MAKE DARK MAiaCS 
K>: 

ERASE 05MPLETELY TC> CHAWOE 



Provider Name: 



Provider Code; 



Instiyctjoos; 

Please lr>dicato the particular 
provider lo which tNs survey 
pertains. Use only one 2*di9ti 
provider code from the bO Ifsled 
on the bacK Ot this page. A 
separate survey Is needed for 
each provider. 



I- L>l 

O 

c 

C; f? 
Q S? 
O' 

>:> 0 

0 © 



II How many of your students 
IdId this provider serve tWs 
1 school year? 


1 What was the start date I 
1 of provider services? 1 
1 Month Dale I 


In >MiIch subjects did your 1 
students receive services 1 
from this provider? 1 


1 

• _ 1-25 
0 26-50 
51-100 
0 Over 100 


' ' October 0 'U* 

Q November 0 0 
' ; 'i December 0 Cf 
0 January 0 0 

0 February 

C Other 0 6 

0 

00 
Off 
0 0 


(; 1 Readlng/tanguage Am only 
0 Mathematics only 
0 Both ReadIng/LA and Math 
0 Uncertain/Oo Not Knoiiv 
n Other 







Are you wiptoyed by the provider ror i^icti you are completing this surve^ ; Yes 



Indicate your response to each of the following items. 



How often does the provlder„. 

I Communicate with you during the school yearT 

Meet the obligations for conducting tutoring sessions? 
I Communicate with teachers during the year? 
Communicate with parents during the year? 




'n\ O 0 ^ 

Cl 0 g Q 

0 g' 'gf gI 

0 "• 0 0 



H 

o 



The provider... 


Strongly Agree 


z 

§: 


-c 

« 

c 

tS 

S 

5 


A» 

V 

a> 

ta 

tft 

Q 


ot 

ar 

tn 

O 

> 

9f 

5 

3 


I Adapted the tutonng seivces to this schoofs cumculum 


0 




r. 




ol 


Integrated the tutoring sen/ices with classroom learning activities 


Q 


0 


6 


O 


o 


Aligned their services with state and local standards. 


Q 


0 


0 


gi 


®l 


Offered services to Special Education and ESL students 


O 


0 


o 


o 


0 


Complied with applicable federal, state, and local laws. 


Ji 


0 


6 


0 






[ 1 believe the services offered by this pnovider positively impacted student achievement 


o 




c 




r|i| 


Overall, 1 am satisfied with the services of this provider 


'1' 


■> 


r* 




*1* 



Additonal comments can be provided on the hack on this form. 



ucuDuic 47d»wMi Awirtn'iVgmbJi^rffii 
MbMrnO ithS^C^'hA^-Ar^aaiM/laMNtrtheNiiv 
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Supplemental Educational Services 
Principal/Site Coordinator Questionnaire 

lOC-S^ fjtfer A, Afcftmatii AJL. i A’n SJ/ 

CmiHTjb* katofcA la StiuiiaSicfkii Fohcy, Tha Liaveraty 
Ali fdgha h^rfdil 



District Name: 


C> Davidson County 
0 Fayette County 


O Hamilton County 0 Knox County 
'• Hardeman County Q Memphis City 


School Nanne 






Naine and Tide ot Parson 
Completing this Survey 











DIRECnONS 

tiUBwa >«wqi-9>AY 



!■ 



VIAKE DARK UAIiKS 
IX 

ERASE 0:'MPL£TELy TO CRAl-IDE 



Provider Name: 



Provider Codr>: 









Instmctiqns; 




Please indicate the particular 




provider to which this survey 




pertains. Use only tine 7-di git 




provider code from the bO llsled 




on the back ot ttiis page. A 




separate survey is needed for 




each provider. 









did ttils provider serve (Ns 
school year? 



',.1-10 
O 11-20 

21-30 
Cl Ovei 30 



! Whal was the start dale 
of provider services? i 


Month 


Dale 


' October 




Q Novembef 


00 


December 


CO 


Q January 


0 0 


'7t February 




0 other 


00 




(Hri 




00 




0'"> 




0 0' 



In tMiich subjects did your 
students receive services 
from this provider? 



O RMdlng/Language Aits oniyl 
O Malhematlcs only 
O Both Raadlng/LA and Main 
O Unc«tBin/Oo Nol Know 
O Ottief 



Are you employed by the provider for which you are completing this survey? ^ Yes C No 



Indicate your response to each of the following items. 




How often does the provider^ 


! Communicate with you during the school year? 






P 






Meet the obligations for conducting tutoring sessions? 




o 


Q 


tit 


G 


Communicate witn teachers during trie year? 








S’ 


I£I 


Communicate with parents during the year? 






r. 


1^1 


0 












0 

« 

Bi 

Sf 

■A 

5 




h 

< 




“C 






>* 




O 




> 




O) 

§ 


i 


tJ 

s 


b 

fO 


9 

S 


The provider... 


C5 




Jl 

ra 


Q 


S 


Adapted the tutoring servces to this schoofs cumculum 




/‘r 




■Ti 


ol 


Integrated the tutoring services with classroom learning activities 


6 


0 


6 




o 


I Aligned their services with state and local standards. 


Q 




Q 


ri 


®l 


Offered services to Special Education and ESL students 


'it 


0 


f? 


0 


i':i 


Complied with applicable federal, state, and local laws. 


1^1 


•? 


6 


rjl 






1 1 twlieve the services offered by this provider positively impacted student achievement 


0 




c 


Q 


1 1 1 


Overall, ( am satisfied with the services of this provider 






r7' 


Ita 





AtldUonal comments can be provided on the hack on this form. 



kiPkiDtjir 

McMf^D 4rftnsfO;ptvAMr*.rjrri<saiv/)aio>T>fvi«*mT 
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Supplemental Educational Services Teacher Questionnaire 

mS^PoHfr A McDoitoli AJ.. <S flow SM 
C'ffiftr fifrJtesforc/nnSducat/onJPo/fc}- Ih* O’fnversii}- p/Mmphu 
a 4 U fiig/iU AesenvJ 

Instructions: According to your school's records, one or more of your students is receiving Supplemental Educational 
Services by a provider approved by the State of Tennessee To meet federal and state compliance regulations, it is 
essential that such services be evaluated. 

PLEASE complete ONE of these brief surveys for EACH provider that is serving one or more of your students 



Dir^cnoNs 



MAKED.VTKMAKKS 
EX era ^ 

ERASE COMPLETELY TO CHAI lOE 



District Name; 



School Mame: I l 

. I 



Piovidor CrvlA: 



G 0 

O 0 



Instri^iqns: 

Please indicate the pailiculai 
provider to which this survey 
pertains. Use only one 2-d)git 
provider code from the 50 listed 
on the back of this page. A 
separate survey is needed loi 
each provider. 



o 

0 

o 

o 

o 

o 

Q 



0 

G- 

0 

Gi 

G 

Q 

G 

G 



How many o( your students 
did This provider serve this 
school year? 


What was the start date I 
of provider services? 1 
Month Dale 1 


In which subjects did your 
students receive services 
from this provider? 


01-10 
11-20 
C 21-30 
Over 30 


O October C G 

G November Q © 
0 December © © 
0 January © © 

0 February © © 

O Other O © 

©0 
oo 

0© 


O Reading/Language Arts only 
Q Mathematics only 
O Both Readtng/LA and Math 
O Uncerlain/Do Not Know 
O Other 







Are you employed by the provider for which you are completing this survey? O Ves O No 


= 1-1 




1 S - ' 


Indicate your response to each of the following items 


£ 1 S 1 

U- G ^ Cl 


How often does the provider... 


i Communicate with you dunng the school year? 


G G O C; 


Meet the obligations tor conducting tutoring sessions? 


O O O O 


Communicate with the principal or school site coordinator dunng the year? 


o o 0 o 



Communicate with parents during the year? O O G O 




5“ « 5 



The provider... 

Adapted the tutoring services to this school's curriculum 
Integrated the tutoring services With classroom learning activities. 

Alig ned ihelr se rv ices with stale and local standards. 

Ottered services to Special Education and ESL students 
Complied with ap plicable federal, state, and local laws 




: Q Q O O I 

G G O Q Q 
G Q O 0~Ql 
C G O C C 
■::! G o G G I 



Overall assessment: 



I believe the services oflered fay this provider positively impacted student achievement. ' Q 

Overall, I am satisfied with the services of this provider G G G C S 

Additional comments can be provided on the back of this fornu 



:■> uwvr^paTYof 

MEMPHIS 



A Termesse^ Board of Regents Institution 
An Equal Opportioufy' • AffirmatiVE Acdon University 
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State of Tonnossoo 

Supplomontal Educational Sorvicoa Parent Questionnaire 

Pottft. ilcDmald AJ„ ill i. M 

Cmt9T for I\»sMroh m Bducobonai Policy. The UnmrstfyofMmphts. 

All Rs^n Re^erveU 



Instructions: 






DlRECnoNS 



IfSCMO iPCV3L»C' 



MAfXOA»BU«IU 

tSi * Oa 

£«A&E COMPLBmV 10 CBANOE 



According to school records, your child is receiving Supplemental Educational Services Please fill In the name 
of your child's provider, district, and school A list of provider names is on the back of this page 



Provider Name: 
District Name; 
School Name; 



Please answer the following survey questions about the services your child has received. 



What was the date the tutoring 
services started this year? 



Q October 2005 
O November 2005 
O December 2005 
0 January 2006 
^ February 2006 
7 Other 2006 



In which subjects did your 
child receive services? 



Q Reading/Language Arts only 
Q Mathematics only 

0 Both Reading/Language Arts and Math 
O Uncertain/Do Not Know 
O Other 



Indicate your response to each of the following items. 




How often d oes the p r ovide r... 



Talk to me about my child’s progress? 






c 


o 


o 


Talk to my child's teachers about his/her progress? 




G 


0 


0 


0 


Send letters or notes home to me about my child's progress? 




G 


c 


Q 


o 


Help my child with subjects s/he is working on In the regular school classroom? 




Q 


Q 


Q 


Q 


Start and end the tutoring sessions at the scheduled time? 




G 


o 


0 


o 












1 


Indicate how much you agree or disagree with each of the following items about 


Oy 






S 


ni 


the provider. 


1 




1 


9 

n 





I believe that the services offered have helped my child's achievement. 
Overall, I am pleased with the services that my child received. 






3 O 



3 OJ 



Indicate how much you agree or disagree with each of the following items about 
the School District. 




I was given information about my child's rights under the No Child Left Behind law. Q Q Q 5 Q 

I was given enough time to decide which service pro vider I wanted for my child. (3 0 Q Q Q 1 

I am pleased with the way my school district helped me obtain Supplemental O 0 G O O 

Educational Services for my child. 



Additonal comments can be provided on the back on this form. 



Do not fold this queshonnairel 



wci •I'vr:'-;."' '.'f 

MEMPHIS 



.*1 Tennessee Board qfFegenls /nstitution 
Ah Bguol OppoflMilfy' • /{ffirmattve Action Unh'enify 



Do not fold this questionnanel 
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Tennessee Supplemental Educational Services 
Provider Questionnaire 

Please make coptes of this form and complete a separate survey tor each district with which you have/had 
a contract to serve students with Supplemental Educationat Services in 2005-2008. 






Please describe the Format oF your services by completing 
the Following items: 



Pmgram durahon (e.Q- academe year) 
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Tennessee Supplemental Educational Services Provider Questionnaire 

(continued) 



II. Information Regarding Students You Served in this District 

1 Complete the table below based on data for tfus distnct tf unable to respond to a particular section provide an explanation in the space piwldetl below 
me table 







#or 


* at 

, Um'rted English 


0Dt 

Students Who 
Achieved Their 


0of 

Students Who 
Made Progress 


Tor i 

Students Who 
Showed Mo 


%0» 

Tutohno 


Grade 


#ofTo(a' 


Special Ed 


Specittc 


' In Their Spectflc 


Improvement 


Sessions 


Level 

1 


Students Served 


Students Served 


Stuoedls Served 


Achievement 

Goals 


Achieved enl 
Goals 


In Their Specific 
Achievement Goai!^ 


Students 

1 




34 Evaluating SES Providers 



Tennessee Supplemental Educational Services Provider Questionnaire 

(continued) 

3, Indicate the number of students served in each curriculum area in which your organization offered services In this district. 
Indicate numbers in all that apply. 



> Reading 


Writing 


1 ? Math 


1 > Other 


K 


K 


K 


K 


1 


1 


1 


1 


2 


2 


2 


2 


3 


3 


3 


3 


A 


4 


4 


4 


5 


5 


5 


5 


6 


6 


6 


6 


7 


7 


7 


7 


8 


8 


8 


8 


9 


9 


9 


9 


10 


10 


10 


10 


11 


11 


11 


11 


12 


12 


12 


12 


III. Provider Perceptions and Activities 



(Indicate your response to each of the following Items as they apply to your staff.) 1 ^ 



1 Tutors communicated with teachers regarding progress of their student(s) C'- Q 




3. Tutors adapted the supplemental services to each school's curriculum ■_ C 

If applicable, briefly dasctibe the process for such adaptations 



Occasionally 
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Tennessee Supplemental Educational Services Provider Questionnaire 

(continued) 



III. Provider Perceptions and Activities (Continued) 

4, Tutors aligned tne supplemerrtal services with the state acadernic content and 
achievement standards 



If applicable, briefly bescnbe the process for such alignments 




5. T utors integrated the tutonng services with classroom learning activities 
If applicable, briefly describe the process for such processes 



6. T Litois showed their lesson plans or malenale used tor tutoring to the homeroom/subject teacher 
of each child they worked with 

If applicable, briefly describe the process for such communications 



7 Tutors gave instruction to students with disabilities, consistent with their Individualized 
Education Plans or Individualized Services under Section 504 of the Rehabilitation Act. 

If ap plicable, briefly describ e the proces s for such activities 



8 T utors protected from public disclosure the identities of all students served and all students 
eligible for services, 

If applicable, bnefly describe the process for such aclrvitres- 



lOceastonaliy 





III. Provider Perceptions and Activities (Continued) 



9 T utors give appropriate inslrucf ion to English Language Learners If it is needed 
If appli c able, briefly describe the process for such activities: 



10 Tutors use appropnate timetables for Improving each student's achievement 
If applicable, bnefly describe the procees for such activities: 



Baseo on your perceptions and expenences, rate the oegree of satisfaction with each of the 
following areas. Use the accompanying ■'Comments" section to elaborate if desired, 
especially where "highly dissatisfied" or "highly satisfied" is indicated. 



$ 



I I 

■fi 5 

S r- 






1 1 Parent cooperation/lnvolvement 



O C' 



Comments 



12 Student attendance 



Comments. 






Mighty Satisfied 





Ill 
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Tennessee Supplemental Educational Services Provider Question mi re 

(continued) 




15. Teacher cooperation/involvement .j i C' O 

Comwents. 



16 Dlstnof cooperation/involvement ? C 



Comments 




1 7 Success at raising student achieveirient to desired levels 
Comments. 






I Highly Satisfied 





Ill 
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Tennessee Supplemental Educational Senrices Provider Questionraire 

(continued) 

Planning, Measuring and Reporting on Progress; 

18 Whal achievement data did you tee to plan, measure, and report on students’ progress for the year? 

Mark all that apply. Please specify where needed 

Tennessee Comprehensive Assessment Program (TCAP) o Reading □ Language DMath 

a Science asocial Studies 



■ *' Gateway Assessment Program 
Specify 




Specrfy 




f 7 I *1 
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Tennessee Supplemental EducatiorKil Services Provider Question mi re 

(continued) 

Overall Impressions 

19 VWiat was the rpost positive outcome or aspect of your work with the district this year? 

I 1 



20 What was the most negative aspect or area in reed of improvement regarding your work with this disthct this year? 



21 Additional Commerits/Recommerxtations 



I 
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Appendix B. Rubric of Overall Provider Effectiveness 



Outcome 


Insufficient 

Information 


Below 

Standards 


Marginal 

Quality 


Acceptable 


Above 

Standards 


1 . student 
Achievement 


There is insuffident 
information 
available to 
determine student 
achievement 
outcomes. 


Students have 
not shown gains 
related to tutoring 
received from 
service providers. 


About half of the 
students have 
made some gain 
related to tutoring 
received from 
service providers. 


There has been 
some gain for the 
majority (over 
60%) of students 
related to tutoring 
received from 
service providers 


The effect size for 
students in the 
provider's program 
is in the top one- 
third of all the effect 
sizes demonstrated 
by providers 
meeting standards 
for student 
achievement. 


2. Communica- 
tion 


There is insuffident 

information 

available to 

determine 

communication 

outcomes. 


Provider has not 
communicated 
with the 
piindpals, 
teachers, and 
parents of 
students served. 


There has 
been limited 
communication 
throughout the 
year between 
the provider and 
at least two of 
the following: 
principals, 
teachers, and 
parents. 


There has been 
some regular 
communication 
throughout the 
yearbetvreen 
the provider and 
the principals, 
teachers, and 
parents of 
students served. 


There is an 
ongoing and 
sustained system 
of communication 
between the 
provider and 
the school- level 
educators as well as 
parents of students 
served. 


3. instructionai 
Pians 


There is insuffident 
information 
available to 
determine 
instructional plans 
of the provider. 


Provider does not 
plan instruction 
explidtly geared 
to student needs 
or to reinforce 
their regular 
academic 
program. 


Provider is in the 
planning stages of 
gearing instruction 
to student needs, 
and reinforcing the 
regular academic 
program. 


Provider has made 
some attempt 
with the majority 
of students to 
plan instruction 
explicitly geared 
to student needs 
and to reinforce 
the regular 
academic program 


Provider 

instructional plans 
are explicitly geared 
to the needs of most 
or all students and 
reinforce the regular 
academic program. 


4. Locai 
and State 
Standards 


There is insuffident 
information 
available to 
determine 
alignment with 
the local and state 
standards. 


None of the 
instructional 
plans used by 
the provider 
are aligned 
with local and 
state academic 
standards for 
students. 


Provider is in the 
process of aligning 
instructional plans 
with local and 
state academic 
standards for 
students. 


Some of the 
instructional 
plans used by 
the provider are 
presently aligned 
with local and 
state academic 
standards for 
students. 


Most or all of 
the instructional 
plans are presently 
aligned with local 
and state academic 
standards for 
students. 


5. Speciai Ed/ 
ELL Students 


There is insuffident 
information 
available to 
determine spedal 
ed/ ELL student 
outcomes. 


Provider 
does not offer 
accommodations 
for addressing the 
needs of special ed 
or ELL students. 


Provider has 
made limited 
accommodations 
for addressing the 
needs of special ed 
and ELL students. 


Provider has 
made some 
accommodations 
for addressing the 
needs of special ed 
and ELL students. 


Provider offers 
appropriate services, 
if needed, to special 
education and ELL 
students. 


6. Provider 
Overail 


There is 
insuffident 
information 
available to 
determine provider 
overall outcomes 


There is overall 
dissatisfaction 
with the provider 
at the district 
and school levels. 


There is more 
dissatisfaction 
than satisfaction 
with the provider 
at the district and 
school levels. 


There are 
mixed but 
mostly positive 
reactions about 
the provider at 
the school and 
district levels. 


There is overall 
satisfaction with 
the provider at the 
district and school 
levels. 










































COMPLIANCE 
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Appendix D. Descriptions of Provider Status Categories 

Full Standing: The provider has demonstrated positive achievement effects, and is fully 

approved to continue services in the following year. 

Remedy requirement: None 

Satisfactory Standing: The provider has demonstrated "indeterminate" achievement effects due to 

insufficient data or equivocal effect sizes, but has shown positive 
implementation outcomes (customer satisfaction, service delivery, compliance). 

The provider has demonstrated positive achievement effects but has minor 
compliance violations. 

Remedy Requirement: Submit improvement plan for correctinq any 
complance violations OR for improvinq student achievement where lack 
of positive evidence is due to equivocal outcomes (neither sufficiently positive 
nor negative effect sizes) not to insufficient sample size. 

The provider has demonstrated "indeterminate" achievement effects due to 
insufficient data or equivocal effect sizes, and has weak or negative 
implementation outcomes. 

Remedy Requirement: Submit improvement plan for improvinq 
implementation quality, any compliance violations, and student achievement 
where lack of positive evidence is due to equivocal outcomes (neither 
sufficiently positive nor negative effect sizes) not to insufficient sample size. 

The provider has demonstrated "negative" achievement effects but has positive 
implementation outcomes. 

Remedy Requirement: Submit improvement plan for improvinq student 
achievement. 

OR, 

The provider was in Probation I status last year, but failed to improve implementation. 

Remedy Requirement: Submit improvement plan for improvinq 
implementation and student achievement. 

Removal: The provider has demonstrated "negative" achievement effects and has 

weak or negative implementation outcomes. 

OR, 

The provider was in Probation II status last year, and although still 
demonstrating positive implementation outcomes, has failed to improve 
student achievement. 

Remedy Requirement: None, but should the current year achievement 
outcomes (to become available in the fall of the next school year) show 
positive effects, the provider may apply for reinstatement after being removed 
for one year. 

NOTE: Separate status determinations will be made for each subject tutored. Thus, it is possible that a provider 
offering tutoring in both reading and mathematics could be classified as Full Standing in one subj ect but Removal 
in the other. 



OR 



Probation I: 



Probation II: 
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For more information please contact: 

CENTER ON INNOVATION & IMPROVEMENT 
121 N. Kickapoo Street 
Lincoln, Illinois 62656 
217-732-6462 
217-732-3696 
www.centerii.org 
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